Signage control system and non-transitory computer-readable recording medium for recording signage control program

ABSTRACT

A signage control system comprises: a primary estimation circuitry to use signage-side images and surveillance-side images to estimate a person feature, attributes and behavior of each person captured in these frame images; a storage device to associate and store results of estimations of the person feature, the attributes and the behavior of each specific person; an estimation result linkage circuitry to use the person feature stored in the storage device to link the results of estimations based on the frame images from multiple ones of the cameras for the same person so as to generate a group of estimation results for each person; and a content change circuitry to change a content displayed on the signage to another based on the attributes of each person expected to be in a position where such person can visually recognize the content on the signage, and based on preceding behavior of such person.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based on and claims the benefit of priority of theprior Japanese Patent Application No. 2020-081542, filed on May 1, 2020,the entire contents of which are incorporated herein by reference.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to a signage control system and anon-transitory computer-readable recording medium for recording asignage control program.

2. Description of the Related Art

Conventionally, there is a video display system called digital signagewhich is placed in a commercial facility such as store, station, and soon. This digital signage is mainly used as an advertising medium, andcan easily change advertisements. The effectiveness of a digital signagewhen placed in a store can be evaluated based on whether “anadvertisement could attract the attention of a customer” or whether “theresult of displaying an advertisement on the digital signage led to aspecific buying behavior”.

The point of whether “an advertisement could attract the attention of acustomer” described above can be analyzed by capturing images of acustomer using a camera mounted on a terminal for the digital signage(hereafter referred to simply as “signage”) when an advertising contentis displayed on the signage, and by using the captured images of thecustomer to obtain information of the customer (line of sight, directionof face, time staring at the signage, and attribute information such asgender and age of the customer) when the content is displayed. In recentyears, a system of tablet-type signage using a tablet terminal is known,which is designed to display an advertising content on (the display of)the signage, and to analyze the attributes and behavior of a customerviewing the content, and further to change the content displayed on thesignage to another depending on the result of the analysis (attributesand behavior of the customer).

Japanese Laid-open Patent Publication 2020-160780 discloses an exampleof such a system (hereafter referred to as “signage control system”) asdescribed above, which is designed to change a content displayed on asignage to another depending on the attributes and behavior of acustomer viewing the content. The system (signage control system) ofthis Japanese Patent Publication estimates the attributes of thecustomer, who is viewing the content displayed on the signage, based onimages of the customer captured by a camera mounted on the signage, andthen, depending on the estimated attributes, the system changes thecontent displayed on the signage to another.

However, conventional signage control systems have the followingproblems. The conventional signage control systems including the systemdisclosed in the above Japanese Patent Publication 2020-160780 aredesigned so that the attributes and behavior of a customer viewing anadvertising content displayed on the signage are analyzed based only onthe images of such customer captured by a camera mounted on the signage,and the content displayed on the signage is changed to another dependingon the result of this analysis. Thus, the conventional signage controlsystems can analyze the behavior of a customer occurring in front of the(camera of the) signage, and can change the content displayed on thesignage to another based on the behavior of the customer, but cannotchange the content displayed on the signage to another considering thebehavior of the customer before the customer comes in front of thesignage. Further, the method, such as the conventional signage controlsystems, which analyzes the attributes and behavior of a customer basedonly on images of the customer captured by a camera mounted on thesignage, cannot start analyzing the attributes and behavior of thecustomer before the customer comes in front of the signage. Therefore,it was not possible to immediately display a content to attract theinterest of the customer on the signage when the customer comes in frontof the signage (and into a place or position in an area where a personcan visually recognize details of the content displayed on the signage).

BRIEF SUMMARY OF THE INVENTION

An object of the present invention is to solve the problems describedabove, and to provide a signage control system and a non-transitorycomputer-readable recording medium for recording a signage controlprogram that make it possible to change a content displayed on a signageto another considering the behavior of a person such as a customerbefore the person such as the customer comes in front of the signage,and also makes it possible to immediately display a content to attractthe interest of the person such as consumer on the signage when theperson such as the customer comes into a position where the person suchas the customer can visually recognize (details of) the contentdisplayed on the signage.

According to a first aspect of the present invention, this object isachieved by a signage control system comprising a signage, asignage-side camera to capture images in front of the signage and atleast one surveillance camera to capture a given capture area, whereinthe signage control system further comprises: a primary estimationcircuitry configured to use signage-side images which are frame imagesfrom the signage-side camera and surveillance-side images which areframe images from the at least one surveillance camera so as to estimatea person feature of each person in these frame images, and also estimateattributes and behavior of the each person captured in these frameimages; a storage device configured to associate and store results ofestimations of the person feature, the attributes and the behavior ofeach specific person as estimated by the primary estimation circuitryusing the frame images from each specific one of the signage-sidecameras and the at least one surveillance camera; an estimation resultlinkage circuitry configured to use the person feature stored in thestorage device to link the results of estimations based on the frameimages from multiple ones of the cameras for the same person so as togenerate a group of estimation results for each person; and a contentchange circuitry configured to change a content displayed on the signageto another based on the attributes of each person who is expected to bein a position where such person can visually recognize the content onthe signage, and also based on preceding behavior of such person beforethen, the attributes and the preceding behavior being contained in thegroup of estimation results generated by the estimation result linkagecircuitry.

According to this signage control system, a content displayed on asignage is changed to another based on the attributes of each person whois expected to be in a position where such person can visually recognizethe content on the signage, and also based on preceding behavior of suchperson before then, all of which are contained in the group ofestimation results generated by linking the estimation results based onthe frame images from multiple ones of the cameras (signage-side cameraand at one surveillance camera) for the same customer. Thus, the contentdisplayed on the signage can be changed to another, considering not onlythe attributes of each person expected to be in a position where suchperson can visually recognize the content on the signage, but also thepreceding behavior of such person before such person has come in frontof the signage (to a position where such person can visually recognizethe content on the signage). Therefore, as compared with theconventional signage control system disclosed in Japanese Laid-openPatent Publication 2020-160780, which changes a content on a signage toanother based only on the attributes and behavior of each personanalyzed based on the frame images of such person captured by asignage-side camera, it is possible to display a content which bettermatches such person in front of the signage (in a position where suchperson can visually recognize the content on the signage).

Further, in contrast to the conventional signage control systemdisclosed in Japanese Laid-open Patent Publication 2020-160780, whichestimates the attributes and behavior of each person from only the frameimages (signage-side images) of such person or other person captured bya signage-side camera, the signage control system of the first aspect ofthe present invention is designed to use not only the signage-sideimages but also frame images (surveillance camera-side images) of eachperson captured by at least one surveillance camera to estimate theperson feature, attributes and behavior of such person captured in theseframe images. Thus, in contrast to the conventional signage controlsystem disclosed in Japanese Laid-open Patent Publication 2020-160780,the process of estimating the attributes, behavior and the like of eachperson who is expected to be in a position where such person canvisually recognize the content on the signage can be started by usingthe surveillance camera-side images having been captured prior to thesignage-side images, before such person comes into the position wheresuch person can visually recognize the content on the signage.Therefore, it is possible to immediately display a content to attractthe interest of such person when such person comes into the positionwhere such person can visually recognize the content on the signage.

According to a second aspect of the present invention, the above objectis achieved by a non-transitory computer-readable recording medium forrecording a signage control program to cause a computer to execute aprocess including the steps of: using signage-side images which areframe images from the signage-side camera and surveillance-side imageswhich are frame images from the at least one surveillance camera so asto estimate a person feature of each person in these frame images, andalso estimate attributes and behavior of the each person captured inthese frame images; associating and storing results of estimations ofthe person feature, the attributes and the behavior of each specificperson using the frame images from each specific one of the signage-sidecameras and the at least one surveillance camera; using the personfeature stored in the storage device to link the results of estimationsbased on the frame images from multiple ones of the cameras for the sameperson so as to generate a group of estimation results for each person;and changing a content displayed on the signage to another based on theattributes of each person who is expected to be in a position where suchperson can visually recognize the content on the signage, and also basedon preceding behavior of such person before then, the attributes and thepreceding behavior being contained in the group of estimation results.

By using a signage control program recorded in the non-transitorycomputer-readable recording medium, it is possible to obtain an effectsimilar to that by the signage control system according to the firstaspect of the present invention.

While the novel features of the present invention are set forth in theappended claims, the present invention will be better understood fromthe following detailed description taken in conjunction with thedrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will be described hereinafter with reference tothe annexed drawings. It is to be noted that the drawings are shown forthe purpose of illustrating the technical concepts of the presentinvention or embodiments thereof, wherein:

FIG. 1 is a schematic block diagram showing an outline configuration ofa signage control system according to an exemplary embodiment of thepresent invention;

FIG. 2 is a schematic block diagram showing an outline hardwareconfiguration of a signage in FIG. 1 ;

FIG. 3 is a schematic block diagram showing an outline hardwareconfiguration of an analysis box in FIG. 1 ;

FIG. 4 is a schematic block diagram showing a functional block diagramof the signage and the analysis box;

FIG. 5 is a schematic block diagram showing a software architecture ofthe signage and a signage management server in FIG. 1 ;

FIG. 6 is a schematic block diagram showing a hardware configuration ofthe signage management server;

FIG. 7 is a flow chart of a content change control process performed bythe signage control system;

FIG. 8 is an explanatory view showing a learning process and aninference process of a DNN (Deep Neural Network) arrival time estimationmodel forming an arrival time estimation circuitry in FIG. 4 ;

FIG. 9 is a flow chart of a content change process of S12 in FIG. 7 ;

FIG. 10 is an explanatory view showing an example of the content changeprocess; and

FIG. 11 is an explanatory view showing a customer behavior trackingprocess and an advanced video analysis achieved by the signage controlsystem.

DETAILED DESCRIPTION OF THE INVENTION

Hereinafter, a signage control system and a signage control programaccording to an exemplary embodiment of the present invention will bedescribed with reference to the drawings. FIG. 1 is a schematic blockdiagram showing an outline configuration of a signage control system 10according to the exemplary embodiment of the present invention. Thepresent embodiment describes an example in which a plurality of signages1 as tablet terminals for digital signage, a plurality of fixed cameras(surveillance cameras) 3 as network cameras (Internet Protocol or IPcameras) for surveillance to capture a given capture area, and ananalysis box 4 connected to these signages 1 and fixed cameras 3 areplaced in a store S such as a chain store.

As shown in FIG. 1 , the signage control system 10 comprises a WiFi AP(WiFi Access Point) 5, a hub 6, a POS (Point Of Sales) register 7 as aPOS system terminal, and a router 8 in addition to the signages 1, thefixed cameras 3 and the analysis box 4 in the store S. Each of thesignages 1 is mainly placed on a product shelf in the store S, andcomprises a touch panel display 14 (refer to FIG. 2 ) on which todisplay advertising contents, for example, for a customer (correspondingto the “person” in the claims) entering the store S. The signage 1further comprises a built-in camera 2 (corresponding to “signage-sidecamera” in the claims) to capture images in front of the signage 1, anduses signage-side images, which are frame images from the built-incamera 2, to perform a recognition process including estimating theattributes (gender and age or generation) and extracting a face vectorof a customer captured in the signage-side images.

The analysis box 4 is connected to each of the signages 1 via the WiFiAP 5 and the hub 6, and also connected to each of the plurality of fixedcameras 3 via a LAN (Local Area Network) and the hub 6 to analyze inputimages from each of these fixed cameras 3. More specifically, theanalysis box 4 subjects input fixed camera-side images (corresponding tothe “surveillance camera-side images” in the claims) which are frameimages from each of the fixed cameras 3 to an object detection process(including face detection process), and also subjects face images of acustomer detected by the object detection process to an inferenceprocess (including an attribute estimation process such as gender andage or generation, a face vector extraction process, a behaviorestimation process, and a person re-identification process tore-identify the customer which will be referred to as “ReID process”).Further, based on the result of attribute estimation, the face vectorand so on sent from the signages 1, the analysis box 4 performs aninference process including the above ReID process and the customerbehavior estimation process. The combination of the analysis box 4 andthe signages 1 corresponds to the “computer” in the claims.

Further, the signage control system 10 comprises a signage managementserver 9 on cloud C. The signage management server 9 is a server placedin a management department (head office or the like) of each storeincluding the store S. A manager of each store, an advertiser of anadvertisement (advertising content) displayed on each signage 1 andother person not only can access the signage management server 9 oncloud C from its own personal computer to know the gender and age of aviewer of the advertising content displayed on the signage 1, and anviewer rating of the advertisement (advertising content), but also canknow a tracking result of the behavior of the customer including whetheror not the customer has contacted with the advertised product afterviewing the advertisement, and whether or not the customer has boughtthe product it has contacted with. Further, the signage control system10 comprises a not shown server of the POS system (POS server) on cloudC.

Next, referring to FIG. 2 , the hardware configuration of theabove-described tablet type signage 1 will be described. The signage 1comprises, in addition to the built-in camera 2, a SoC(System-on-a-Chip) 11, a touch panel display 14, a speaker 15, a memory16 for storing various data and programs, a communication unit 17, arechargeable battery 18, and a charging terminal 19. The SoC 11comprises a CPU (Central Processing Unit) 12 configured to control theentire device and perform various operations, and a GPU 13 used for, forexample, inference processes in various learned DNN (Deep NeuralNetworks) models.

The programs stored in the memory 16 include a signage-side controlprogram 50 including various inference models included in an AI(Artificial Intelligence) model group 51 described later in FIG. 5 . Thecommunication unit 17 comprises a communication IC (Integrated Circuit)and an antenna. The signage 1 is connected to the analysis box 4 and thesignage management server 9 on cloud C via the communication unit 17 anda network. The rechargeable battery 18 is a battery such as lithium-ionbattery which can be used repeatedly by charging, and stores power froma commercial power supply after converting it to DC power by an AC/DCconverter, and further supplies it to respective parts of the signage 1.

Next, referring to FIG. 3 , the hardware configuration of the analysisbox 4 will be described. The analysis box 4 comprises: a CPU 21configured to control the entire device and perform various operations;a hard disk 22 for recording or storing various data and programs; a RAM(Random Access Memory) 23; inference chips (hereafter referred to simplyas “chips”) 24 a to 24 h as DNN (Deep Neural Networks) inferenceprocessors; and a communication control IC 25. The CPU 21 is a commongeneral-purpose CPU or a CPU designed to increase parallel processingperformance to process a lot of video streams at the same time. Further,the data recorded or stored in the hard disk 12 include video data(fixed camera-side images) obtained by decoding (data of) video streamsinput from each of the fixed cameras 3, and also include results ofestimations using a primary estimation circuitry 32 of the analysis box4 and a primary estimation circuitry 41 of the signage 1 which aredescribed later. Further, the programs recorded or stored in the harddisk 22 include an analysis box OS (Operating system) program, andlearned DNN inference models (learned DNN models for various inferenceprocesses) for such as face detection process, attribute (gender and ageor generation) estimation (face recognition) process, face vectorextraction process, behavior estimation process, ReID process, andlater-described arrival time estimation process. The above-describedlearned DNN models for various inference processes together with thesignage-side control program 50 described later in FIG. 5 form a“signage control program” in the claims. The combination of the harddisk 22 and the memory 16 of the signage 1 described above correspondsto the “non-transitory computer-readable recording medium” in theclaims.

The (inference) chips 24 a to 24 h are preferably processors optimizedfor DNN inference (chips dedicated for the inference), but can begeneral-purpose GPUs (Graphics Processing Units) used for common use, orother processors. Further, the chips 24 a to 24 h can be devices made byintegrating (mounting) a plurality of chips (inference processors) onone board computer. It is also possible to mount multiple kinds of chipson one analysis box 4. As shown in FIG. 3 , the (inference) chips 24 ato 24 h are connected to the CPU 21 by PCI (Peripheral ComponentInterconnect) Express or USB (Universal Serial Bus). Note that it ispossible to connect a part of the chips 24 a to 24 h to the CPU 21 byPCI Express, and connect the other chips to the CPU 21 by USB. Further,the communication control IC 25 has a LAN port 26 which is a port forconnection to LAN based on the Ethernet Standard.

FIG. 4 shows functional blocks of the signage 1 and the analysis box 4described above. As the functional blocks, the analysis box 4 comprisesa video input circuitry 31, a primary estimation circuitry 32, a storagedevice 33 (corresponding to the hard disk 22 in FIG. 3 ), an arrivaltime estimation circuitry 34, an estimation result linkage circuitry 35,a content change circuitry 36 and a cooperative processing circuitry 37.The video input circuitry 31 is formed by the communication control IC25 and the CPU 21 in FIG. 3 , and is configured to receive and decode(data of) video streams input from each of the fixed cameras 3 into dataof frame images (fixed camera-side images). The primary estimationcircuitry 32 is configured to use the fixed camera-side images to infer(estimate): a face vector (corresponding to the “person feature” in theclaims) to identify each customer in the fixed camera-side images;attributes (gender and age or generation) of each customer captured inthe fixed camera-side images; and behavior of each customer captured inthe fixed camera-side images. The behavior of each customer at leastincludes contact of such customer with a product (that the customertakes the product in hand). The behavior of each customer describedabove may include behavior such that such customer is walking, suchcustomer is looking at the signage 1 placed on a product shelf, and soon.

The storage device 33 is configured to associate and store the resultsof estimations of the face vector, the attributes and the behavior ofeach specific customer as estimated by the primary estimation circuitry32 using the frame images from each specific one of the fixed cameras 3.The storage device 33 is also configured to associate and store the facevector and the results of estimations of the attributes and behavior ofeach specific customer as estimated by a signage 1-side primaryestimation circuitry 41 described later using frame images (signage-sideimages) from a built-in camera 2 of such signage 1. Here, the facevector, the attributes and the behavior stored in the storage device 33described above are those respectively obtained by the inference(estimation) performed by the primary estimation circuitry 32 for eachspecific customer during the time from frame-in to frame-out of suchspecific customer in the images captured by the specific camera (fixedcamera 3 or built-in camera 2), more specifically, from when the captureof such specific customer in the images starts to when the capture ofsuch specific customer in the images ends.

The arrival time estimation circuitry 34 is configured to estimatearrival time of each customer captured in the frame images captured bythe fixed camera 3 at which such customer is expected to arrive at aposition where such customer can visually recognize a content displayedon each signage 1. More precisely, the arrival time estimation circuitry34 is configured so that, from a motion vector of such customer capturedin the fixed camera-side images and from the time point at which suchcustomer appears in the fixed camera-side images, the arrival timeestimation circuitry 34 estimates the arrival time of such customer atwhich such customer is expected to arrive at the position where suchcustomer can visually recognize the content displayed on such signage 1.

Based on the face vector of each customer stored in the storage device33, the estimation result linkage circuit 35 links the results ofestimations (results of estimations of the face vector, the attributesand the behavior) based on the frame images from multiple ones of thecameras (built-in cameras 2 of the plurality of signages 1 and theplurality of fixed cameras 3) for the same customer so as to generate agroup of estimation results for each customer. More precisely, theestimation result linkage circuitry 35 is configured so that, based onthe face vector stored in the storage device 33 for each customer who isexpected to arrive at a position where such customer can visuallyrecognize a content on each signage 1 as a result of the estimationusing the arrival time estimation circuitry 34, the estimation resultlinkage circuitry 35 links the estimation results based on the frameimages from multiple ones of the cameras (built-in cameras 2 of thesignages 1 and fixed cameras 3) for the same customer so as to generatea group of estimation results for each customer. The estimation resultlinkage process using the estimation result linkage circuitry 35 isperformed using a DNN model for the re-identification process for eachcustomer (customer ReID process based on the face vector of eachcustomer) included in the learned DNN models for various inferenceprocesses stored in the hard disk 22.

The group of estimation results generated using the estimation resultlinkage circuitry 35 includes the attributes of a customer who isexpected to be in a position where such customer can visually recognizea content displayed on the signage 1, and also includes the (preceding)behavior of such customer before then. Based on such attributes and suchpreceding behavior of the customer, the content change circuitry 36changes the content displayed on (the touch panel display 14 of) thesignage 1 to another. The cooperative processing circuitry 37 isconfigured to perform a process to receive, from the signage 1, variousestimation results (face vector, attributes and behavior of thecustomer), a tracking ID described later, and so on, and store them inthe storage device 33, and also perform a process to send, to thesignage 1, identification information such as URL (Uniform ResourceLocator) of a content to be displayed which is output from the contentchange circuitry 36. Among the functional blocks of the analysis box 4described above, the primary estimation circuitry 32, the arrival timeestimation circuitry 34 and the estimation result linkage circuitry 35are formed by the CPU 21 and the (inference) chips 24 a to 24 h (referto FIG. 3 ). Further, the storage device 33 is formed by the hard disk22 in FIG. 3 , and the content change circuitry 36 is formed by the CPU21, while the cooperative processing circuitry 37 is formed by thecommunication control IC 25 and the CPU 21 in FIG. 3 .

As the functional blocks, the signage 1 comprises a video inputcircuitry 40, a primary estimation circuitry 41, a cooperativeprocessing circuitry 42 and a content display control circuitry 43 inaddition to the built-in camera 2, the touch panel display 14 and thespeaker 15 described above. The video input circuitry 40 is mainlyformed by the SoC 11 in FIG. 2 (including a not shown I/O orInput/Output chip set), and receives and decodes (data of) the videostreams input from the built-in camera 2 of each signage 1 into frameimage data (signage-side images). The primary estimation circuitry 41 isconfigured to perform, based on the signage-side images described above,a process similar to that of the primary estimation circuitry 32 in theanalysis box 4. More specifically, based on the signage-side images, theprimary estimation circuitry 41 infers (estimates) a face vector(corresponding to the “person feature” in the claims) for identifying acustomer in the signage-side images, and also infers (estimates) theattributes (gender and age or generation) and behavior of the customercaptured in the signage-side images.

The cooperative processing circuitry 42 is configured to perform aprocess to send the estimation results obtained by using the primaryestimation circuitry 41 to the analysis box 4, and a process to receiveidentification information of the content to be displayed which isoutput from the content change circuitry 36, and output the receivedidentification information to the content display control circuitry 43.The content display control circuitry 43 is configured to control tooutput an image and a sound of a content corresponding to theidentification information (such as URL) of the content output from thecooperative processing circuitry 42 to the touch panel display 14 andthe speaker 15, respectively. The video input circuitry 40, the primaryestimation circuitry 41 and the content display control circuitry 43among the functional blocks of the signage 1 described above are formedby the SoC 11 in FIG. 2 , while the cooperative processing circuitry 42is formed by the communication unit 17 and (mainly the CPU 12) of theSoC 11 in FIG. 2 .

FIG. 5 shows a software architecture of the signage 1 and the signagemanagement server 9 in FIG. 1 . The signage 1 stores, in the memory 16(refer to FIG. 2 ), a signage-side control program 50 and an Android OS(mobile operating system) 54 shown in FIG. 5 . The signage-side controlprogram 50 is mainly formed by AI models 51 consisting of variousinference models, a content viewer 53 which is a viewer for videocontents, and a contents management program 52 which is a kind ofso-called CMS (Contents Management System). The AI models 51 include aface detection model 51 a, a face recognition (gender/age estimation)model 51 b, a vectorization model 51 c, a product contact determinationmodel 51 d, a person detection model 51 e, and so on.

The face detection model 51 a is configured to detect a face of acustomer captured in signage-side images input from the built-in camera2 so as to output coordinate position of the detected face (for example,coordinate representing the center of the face and coordinate arearepresenting the horizontal width and vertical width of the face). Theface recognition (gender/age estimation) model 51 b is configured sothat, if the face of the customer detected using the face detectionmodel 51 a is suitable for the recognition of the attributes of thecustomer (for example, if the detected face of the customer isfront-facing, and if, at the same time, such face has some sufficientsize), the face recognition model 51 b uses a cut-out image of the faceof the customer to perform an estimation process of the attributes(gender and age or generation) of the customer. Further, thevectorization model 51 c is configured to perform a process to vectorizethe cut-out image of the face (face image) (detected by the facedetection model 51 a) described above to obtain a vector, and save(store) the thus obtained vector in the memory 16 as a face vector(corresponding to the “person feature” in the claims).

The person detection model 51 e is configured to detect customerscaptured in the signage-side images input from the built-in camera 2.The product contact determination model 51 d is configured so that,based on the skeleton information of each customer captured in eachsignage-side image as detected by the person detection model 51 e, theproduct contact determination model 51 d determines the posture of eachcustomer in front of the product shelf on which each signage 1 isplaced, and based on this posture, determines whether or not eachcustomer has contacted with a product (that the customer has taken theproduct in hand). Note that the person detection model 51 e is also usedfor a process to count viewers of each signage 1 placed on a productshelf (to count the number of customers, whose line of sight or face isdirected to the signage 1, among the customers captured in thesignage-side images), and further used for consumer rating survey of thesignage 1 described later.

Further, referring to FIG. 5 , the signage management server 9 stores asignage management program 56, a dashboard 57 and a portal 58 in a harddisk 62 (refer to FIG. 6 ). The signage management program 56 is aprogram to manage each signage 1 in the signage control system 10. Thedashboard 57 is a software to aggregate and visualize the statisticalinformation on results of tracking the behavior of each customer,including the attributes (gender and age or generation) and stay time ofthe viewer (customer) of the advertising content displayed on (the touchpanel display 14 of) each signage 1, and also including whether or notthe customer has touched a product after looking at the advertisingcontent, and whether or not the customer has bought the product aftercontacting with the product. The portal 58 is a kind of so-calledenterprise portal (software designed so that in order to effectivelysearch and use various information, applications and the like scatteredthroughout an enterprise, the software integrates and displays theseinformation, applications and the like on the screen of a computer).Applications which can be accessed from the portal 58 includeapplications to set each advertising content desired to be displayed onthe touch panel display 14 of the signage 1, and to also set displayconditions for these advertising contents.

Next, referring to FIG. 6 , the hardware configuration of the signagemanagement server 9 will be described. The signage management server 9comprises a CPU 61 configured to control the entire device and performvarious operations, a hard disk 62 configured to store various data andprograms, a RAM (Random Access Memory) 63, a display 64, an operationunit 65 and a communication unit 66. The programs stored in the harddisk 62 include the signage management program 56, the dashboard 57, and(programs for) the portal 58 that are described above.

Next, referring the flow chart of FIG. 7 , an outline of a contentchange control process performed by the signage control system 10 of thepresent exemplary embodiment will be described. First, the signage1-side primary estimation circuitry 41 and the analysis box 4-sideprimary estimation circuitry 32 detect faces of customers (face images)from the frame images (signage-side images) input from the built-incamera 2 of the signage 1 and the frame images (fixed camera-sideimages) input from the fixed camera 3, respectively (S1 and S2). Notethat the face detection process performed by the signage 1-side primaryestimation circuitry 41 uses the face detection model 51 a describedabove (refer to FIG. 5 ), while the face detection process performed bythe analysis box 4-side primary estimation circuitry 32 uses the DNNmodel for face detection process described above, which is included inthe learned DNN models for various inference processes stored in thehard disk 22.

To describe the processes from S3 onward, signage 1-side processes andanalysis box 4-side processes will be described separately. First, thesignage 1-side processes will be described. When the face detectionprocess in S2 is completed, the signage 1-side primary estimationcircuitry 41 assigns a tracking ID to each of the faces detected in S2(S3). More specifically, based on the time point at which each of thesignage-side images was captured by the same built-in camera 2, andbased on the coordinate position of the face (or the coordinate positionand size of the face) detected by the face detection model 51 a fromeach of these signage-side images, the signage 1-side primary estimationcircuitry 41 assigns the same tracking ID to (the face of) the samecustomer over the frames so as to perform a tracking process ofcustomers captured by the same built-in camera 2.

Then, if the signage 1-side primary estimation circuitry 41 detects, forthe first time, a suitable face for the recognition of the attributes ofthe customer (for example, if (an image of) a face is detected such thatthe face is front-facing, and, at the same time, the face has somesufficient size) from the faces assigned with a specific tracking ID,the signage 1-side primary estimation circuitry 41 cuts out the image ofthe face (face image) (suitable for the attribute recognition) from theframe images (signage-side images) serving as a source for detection(S4). Subsequently, the signage 1-side primary estimation circuitry 41uses the face recognition (gender/age estimation) model 51 b describedabove to estimate the attributes (gender and age or generation) of suchcustomer based on the face image (S5).

Further, the signage 1-side primary estimation circuitry 41 uses thevectorization model 51 c described above to perform a process tovectorize the face image to obtain a face vector (corresponding to the“person feature” in the claims) (S6). In addition, the signage 1-sideprimary estimation circuitry 41 uses the product contact determinationmodel 51 d described above and the like to estimate the behavior of eachcustomer in front of a product shelf on which the signage 1 is placed,including whether or not such customer has contacted with a product(that the customer has taken the product in hand) (S7). Note that atleast when the later-described DNN arrival time estimation model learns,the signage 1-side primary estimation circuitry 41 performs a process toobtain motion tracking of each customer (combinations of the centerpoints and time points of bounding boxes for each customer) captured inthe signage-side images captured by the built-in camera 2.

The signage 1-side cooperative processing circuitry 42 is configured tosend, to the analysis box 4, the estimation results using the primaryestimation circuitry 41 described above, more specifically, theestimation results of the face vector, the attributes, the (customer)behavior, the tracking ID and the motion tracking for each specificcustomer based on the frame images from the built-in camera 2 of eachspecific signage 1. The analysis box 4-side cooperative processingcircuitry 37 is configured to receive the various estimation resultsfrom the signage 1 (the face vector, attributes, behavior, tracking IDand motion tracking of such specific customer), and then associate andstore, in the storage device 33, these face vector, attributes,behavior, tracking ID and motion tracking of such specific customer asestimated based on the frame images from the built-in camera 2 of suchsignage 1 (S8).

Next, the analysis box 4-side processes will be described. When the facedetection process in S2 is completed, the analysis box 4-side primaryestimation circuitry 32 assigns a tracking ID to each of the facesdetected in S2 (S3). More specifically, based on the time point at whicheach of the fixed camera-side images was captured by the same fixedcamera 3, and based on the coordinate position of the face (or thecoordinate position and size of the face) detected by the DNN model forface detection process (stored in the hard disk 22) described above fromeach of these fixed camera-side images, the analysis box 4-side primaryestimation circuitry 32 assigns the same tracking ID to (the face of)the same customer over the frames so as to perform a tracking process ofcustomers captured by the same fixed camera 3.

Then if, like the signage 1-side primary estimation circuitry 41described above, the analysis box 4-side primary estimation circuitry 32detects, for the first time, a suitable face for the recognition of theattributes of the customer from the faces assigned with a specifictracking ID, the analysis box 4-side primary estimation circuitry 32cuts out the face image from the frame images (fixed camera-side images)serving as a source for detection (S4). Subsequently, the analysis box4-side primary estimation circuitry 32 uses the DNN model for attributeestimation (face recognition) process (stored in the hard disk 22)described above to estimate the attributes (gender and age orgeneration) of such customer based on the face image (S5). Further, theanalysis box 4-side primary estimation circuitry 32 uses the DNN modelfor face vector extraction (stored in the hard disk 22) described aboveto perform a process to vectorize the face image to obtain a face vector(S6).

In addition, the analysis box 4-side primary estimation circuitry 32uses the DNN model for behavior estimation process (stored in the harddisk 22) described above to estimate the behavior of each customer(customer behavior) captured in the fixed camera-side images (S7). Notethat the analysis box 4-side primary estimation circuitry 32 alsoperforms a process to obtain motion tracking of each customer(combinations of the center points and time points of bounding boxes foreach customer) captured in the fixed camera-side images captured by eachfixed camera 3, and from this motion tracking of each customer, obtain amotion vector (refer to FIG. 8 ) of each customer captured by each fixedcamera 3. The analysis box 4-side storage device 33 associates andstores the estimation results using the primary estimation circuitry 32described above, more specifically, the face vector, the attributes, the(customer) behavior, the tracking ID, the motion tracking and the motionvector for each specific customer as estimated based on the frame imagesfrom each specific fixed camera 3 (S8).

Next, from a motion vector of each customer captured in the fixedcamera-side images, and from a time point at which the each customerappears in the fixed camera-side images, the arrival time estimationcircuitry 34 of the analysis box 4 estimates arrival time of the eachcustomer at which such customer is expected to arrive at a positionwhere such customer can visually recognize a content displayed on eachsignage 1 (S9). For example, if the arrival time estimation circuitry 34is implemented by using a learned DNN arrival time estimation model(learned DNN model for arrival time estimation), the arrival time of theeach customer, at which such customer is expected to arrive at theposition where such customer can visually recognize the contentdisplayed on each signage 1, can be estimated by inputting the motionvector (motion vector of the each customer captured by each fixed camera3) obtained by the primary estimation circuitry 32 as described above tothe learned DNN model for arrival time estimation.

The learning of the DNN arrival time estimation model is done asfollows. Among the analysis box 4-side functional blocks (refer to FIG.4 ) in the signage control system 10 of the present exemplaryembodiment, the arrival time estimation circuitry 34, the estimationresult linkage circuitry 35 and the content change circuitry 36 aredisabled in function (not used), while the signage 1-side primaryestimation circuitry 41 and the analysis box 4-side primary estimationcircuitry 32 are used to collect motion tracking data of each customercaptured in the frame images captured by the built-in camera 2 and thefixed camera 3. Here, the above-described motion tracking data of eachcustomer means combinations of the center points and time points ofbounding boxes for each customer, an example of which is shown in FIG. 8where reference numerals 70 a to 70 g represent the center points ofbounding boxes for such customer.

From the thus collected motion tracking data and motion vectors 71 foreach customer for a given time period, the analysis box 4-side primaryestimation circuitry 32 generates combinations of: the motion vectors 71for each customer captured by each fixed camera 3; a time point at whichsuch customer appears in the images captured by such fixed camera 3(hereafter referred to as “time point of appearance in the fixed camera3”); and a time point at which such customer appears in the imagescaptured by the built-in camera 2 of each signage 1 (hereafter referredto as “time point of appearance in the signage 1”). Note that in theexample shown in FIG. 8 , the “time point of appearance in the fixedcamera 3” is a time point corresponding to the center point 70 c at thearrow end of the motion vector 71 among the center points of thebounding boxes of the customer appearing in the images captured by thefixed camera 3.

Then, the DNN model to estimate a time period T from the “time point ofappearance in the fixed camera 3” to the “time point of appearance inthe signage 1” (namely the above-described DNN arrival time estimationmodel) is allowed by the CPU 21 of the analysis box 4 to learn using thecombined (aggregated) data of the motion vectors 71, the “time point ofappearance in the fixed camera 3” and the “time point of appearance inthe signage 1” as learning data. The hard disk 22 includes a number ofsuch DNN arrival time estimation models equal to the number ofcombinations of the fixed cameras 3 and the signages 1 in the store S.For example, if the number of fixed cameras 3 is 3, and the number ofsignages 1 is 4, the DNN models stored in the hard disk 22 include 12(12 kinds of) DNN arrival time estimation models.

When each learned DNN arrival time estimation model generated by thelearning described above infers, the arrival time estimation circuitry34 of the analysis box 4 inputs the motion vectors 71 obtained by theanalysis box 4-side primary estimation circuitry 32 to the each learnedDNN arrival time estimation model so as to obtain the time period T(time period from a time point at which a customer appears in an imagecaptured by a specific fixed camera 3 to a time point at which suchcustomer appears in an image captured by the built-in camera of aspecific signage 1). Thereafter, the arrival time estimation circuitry34 of the analysis box 4 adds the time period T to the time point atwhich such customer appears in the image captured by the specific fixedcamera 3 (fixed camera-side image), so as to estimate arrival time ofsuch customer at which such customer is expected to arrive at a positionwhere such customer can visually recognize a content displayed on thespecific signage 1 (estimate time point at which such customer isexpected to appear in the images captured by the built-in camera 2 ofthe specific signage 1).

When the signage control system 10 is operated (when each learned DNNarrival time estimation model described above infers), the arrival timeestimation circuitry 34 of the analysis box 4 uses each of the learnedDNN arrival time estimation models (which are equal in number to thecombinations of the fixed cameras 3 and the signages 1) to estimatearrival time at which each customer appearing in the images captured byeach fixed camera 3 is expected to arrive at a position where suchcustomer can visually recognize a content displayed on each signage 1.Based on these estimation results, the arrival time estimation circuitry34 predicts a person who, at a specific time point, is expected to be inthe position where such person can visually recognize the contentdisplayed on the each signage 1.

Now, referring back to FIG. 7 , the processes, after S9 above, performedby the signage control system 10 will be described. When the arrivaltime estimation process in S9 above is completed, the estimation resultlinkage circuitry 35 of the analysis box 4 uses the face vector and thetracking ID stored in the storage device 33 for each customer, who isexpected to arrive at a position where such customer can visuallyrecognize a content on each signage 1 as a result of the estimationusing the arrival time estimation circuitry 34, to link the estimationresults based on the frame images from multiple ones of the cameras(built-in cameras 2 of the signages 1 and the fixed cameras 3) for thesame customer so as to generate a group of estimation results for eachcustomer (S10). As described above, the estimation result linkageprocess using the estimation result linkage circuitry 35 is performedusing a DNN model for the re-identification process for each customer(customer ReID process). Thus, this signage control system 10 can startthe re-identification process for each customer (customer ReID process)before each customer arrives at a position where such customer canvisually recognize a content on each signage 1. Note that the DNN modelfor the customer re-identification process is a learned DNN model whichre-identifies the same customer captured in the frame images frommultiple ones of the cameras (built-in cameras 2 of the signages 1 andthe fixed cameras 3) and assigns the same global ID (customer ID overthe cameras) to the same customer.

When the process of generating the group of estimation results in S10above is completed, the content change circuitry 36 of the analysis box4 operates so that if a customer is expected to be in the position, at aspecific time point, where such customer can visually recognize acontent on the signage 1 (more precisely, if it is estimated that thereis a customer, at a specific time point, who is expected to have arrivedat a position where such customer can visually recognize a content onthe signage 1, and who is also expected to be in the position for apredetermined time or longer where such customer can visually recognizethe content on the signage 1) as a result of the estimation using thearrival time estimation circuitry 34 (YES in S11), the content changecircuitry 36 changes the content displayed on (the touch panel display14 of) the signage 1 to another (S12), based on the data contained inthe group of estimation results generated by the estimation resultlinkage circuitry 35, more specifically, based on the attributes (genderand age or generation) of such customer who is expected to be in theposition where such customer can visually recognize the contentdisplayed on the signage 1, and also based on the preceding behavior ofsuch customer before then.

In other words, for each customer who is expected to arrive at aposition where such customer can visually recognize a content on thesignage 1 as a result of the estimation using the arrival timeestimation circuitry 34, the content change circuitry 36 of the analysisbox 4 operates so that, at a time point based on the estimated arrivaltime of such customer at which such customer is expected to arrive atthe position where such customer can visually recognize the content onthe signage 1 as estimated by the arrival time estimation circuitry 34(for example, at the time point of the estimated arrival time itself, orat a time point a predetermined time after the estimated arrival time),the content change circuitry 36 changes the content displayed on thesignage 1 to another based on the attributes and the preceding behaviorof such customer which are contained in the group of estimation resultsgenerated by the estimation result linkage circuitry 35. Note that ifthere is a variation in the attributes of a customer having arrived at aposition where such customer can visually recognize a content on thesignage 1, such attributes being contained in the group of estimationresults described above (more specifically, if all the attributes ofsuch customer estimated based on the frame images from the built-incameras 2 of the signages 1 and the attributes of such customerestimated based on the frame images from the fixed cameras 3 are notcompletely the same), then the content change circuitry 36 changes thecontent displayed on the signage 1 to another based on the most likely(the most numerous) attributes among these attributes.

Next, referring to the flow chart of FIG. 9 , the content change processof S12 in FIG. 7 above will be described in detail. If, as a result ofthe estimation using the arrival time estimation circuitry 34, thenumber of customers, at a specific time point, who are expected to be ina position where such customers can visually recognize a content on thesignage 1 (more precisely, customers at a specific time point who areexpected to have arrived at the position where such customers canvisually recognize the content on the signage 1, and who are alsoexpected to be in the position for a predetermined time or longer wheresuch customers can visually recognize the content on the signage 1) isdetermined to be one (YES in S21), the content change circuitry 36 ofthe analysis box 4 operates so that, at a time point based on theestimated arrival time of such customer at which such customer isexpected to arrive at the position where such customer can visuallyrecognize the content on the signage 1, the content change circuitry 36changes the content displayed on the signage 1 to another based on theattributes (gender and age or generation) and the preceding behavior ofsuch customer (preceding behavior of such customer before then,including contact of such customer with products, viewing time of suchcustomer to view such signage 1, viewing time of such customer to view asignage 1 placed on another product shelf, and so on) (S22).

In other words, the content change circuitry 36 of the analysis box 4operates so that at a time point based on the estimated arrive time ofsuch customer at which such customer is expected to arrive at theposition where such customer can visually recognize the content on thesignage 1 (for example, at the time point of the estimated arrival timeitself, or at a time point a predetermined time after the estimatedarrival time), the content change circuitry 36 changes the content onthe signage 1 to another which is considered to match such customer,considering the attributes and the preceding behavior of such customerwho is expected to be in the position where such customer can visuallyrecognize the content on the signage 1.

For example, as shown in FIG. 10 , if, at a specific time point, acustomer who is a woman in her thirties is expected to arrive at aposition where such customer (she) can visually recognize a content on aspecific signage 1, and if, at such time point, no other person is inthe position where such person can visually recognize the content onsuch signage 1, then the content change circuitry 36 of the analysis box4 operates so that at a time point based on the estimated time point atwhich such customer (she) is expected to arrive at the position wheresuch customer (she) can visually recognize the content on the signage 1,the content change circuitry 36 changes the content displayed on thesignage 1 to another based on the attributes of such customer (woman inher thirties) and on the preceding behavior of such customer (her) (forexample, viewing time of such customer to view the signage 1) (S22).

More specifically, for example, as shown in FIG. 10 , the content changecircuitry 36 of the analysis box 4 operates so that at a time point when5 seconds of viewing by such customer to view such signage 1 has passed(at a time point when the behavior of such customer to view such signage1 has continued for 5 seconds after the arrival time of such customer),the content change circuitry 36 changes the advertising contentdisplayed on the touch panel display 14 of such signage 1 from a generaladvertisement to a specific advertisement A which matches or correspondsto the attributes of such customer (woman in her thirties). Thus, onlyby placing a signage 1 on a product shelf or the like in the store S,the signage control system 10 of the present exemplary embodimentenables interactive advertising display (real-time advertising displaychange matching or corresponding to the attributes and the precedingbehavior, such as viewing time, of the viewer or customer) linked to thecontents management program 52 (refer to FIG. 5 ) which is a kind ofCMS.

Further, the CPU 61 (refer to FIG. 6 ) of the signage management server9 collects information of the attributes and behavior of each customerwho is expected to be in a position where such customer can visuallyrecognize a content on each signage 1 as described above, information ofviewer rating of each signage 1, and other information so as to analyzethese information and store the analysis results in the hard disk 62. Amanager of each store, an advertiser of an advertisement (advertisingcontent) displayed on each signage 1 and other person can access (thedashboard 57 of) the signage management server 9 (refer to FIG. 5 ) fromits own personal computer 81 so as to check and use these information(analysis results) including the attributes and behavior of eachcustomer and the viewer rating of each signage 1 described above asshown in FIG. 10 . Here, the viewer rating of each signage 1 describedabove means a ratio of the number of customers with the line of sight orface directed to each signage 1 to the total number of customerscaptured in the signage-side images of each signage 1 (customers whohave passed in front of each signage 1). Note that the personal computer81 of each of the manager, the advertiser of the advertisement(advertising content) and other person can be designed to receive,through a data distribution API (Application Programming Interface), theanalysis results of each customer described above, such as theattributes and behavior, stored in the hard disk 62 of the signagemanagement server 9.

Referring back to FIG. 9 , if the number of customers, at a specifictime point, who are expected to be in the position where such customerscan visually recognize the content on the signage 1 is determined in S21above to be plural (NO in S21), the content change circuitry 36 of theanalysis box 4 determines whether or not, in the attributes estimated bythe signage 1-side primary estimation circuitry 41 or the analysis box4-side primary estimation circuitry 32 for these plural customers, thereis a common attribute for all these plural customers (S23). If there isa common attribute for all the plural customers as a result of thedetermination of S23 above (YES in S23), the content change circuitry 36of the analysis box 4 displays, on the touch panel display 14 of thesignage 1, a content matching or corresponding to such common attributeat a time point based on the specific time point described above (timepoint at which the plural customers are expected to be in the positionwhere they can visually recognize the content on the signage 1) (S24).Note that the time point based on the specific time point describedabove can be the specific time point itself, or a time point apredetermined time (for example, 5 seconds) after the specific timepoint. On the other hand, if a common attribute for all the pluralcustomers is absent as a result of the determination of S23 (NO in S23),the content change circuitry 36 of the analysis box 4 displays apredetermined standard (default) content on the touch panel display 14of the signage 1 (S25).

As shown in FIG. 11 , in the signage control system 10 of the presentexemplary embodiment, the analysis box 4 is used as if it were a hub forthe plurality of signages 1 and the plurality of fixed cameras 3 placedin store S so as to enable advanced video analysis by merging theresults of analyzing the images captured by (the built-in cameras 2 of)the signages 1 and the fixed cameras 3 (merging the estimation resultsof the attributes, behavior and the like of each customer). In order tomerge the analysis results of the captured images by the signages 1 andthe fixed cameras 3 in the present signage control system 10, theestimation result linkage circuit 35 is used for re-identifying eachcustomer based on its face vector (customer ReID process) so as to linkthe estimation results (analysis results) based on the frame images frommultiple ones of the cameras (built-in cameras 2 of the signages 1 andthe fixed cameras 3) for the same customer.

Thus, the present signage control system 10 can track not only theattributes (gender and age or generation) of each customer havingarrived at a position where such customer can visually recognize acontent on the signage 1, and the viewer rating of each advertisingcontent displayed on the signage 1 by such customers, but also thebehavior of such customer, including whether or not such customer hascontacted with a product after viewing the advertising content displayedon the signage 1, and whether or not such customer has bought theproduct it has contacted with. Further, a manager of each store, anadvertiser of the advertising content displayed on the signage 1 andother person can access the signage management server 9 on cloud C fromits own personal computer 81 to check (see) (information of) thetracking result of the behavior of such customer, including whether ornot such customer has contacted with the product after viewing theadvertisement (advertising content) described above, and whether or notsuch customer has bought the product it has contacted with. As shown inFIG. 11 , an example of the (information of) the tracking result of thebehavior of such customer is “that a woman in her thirties viewed (theadvertising content of) the signage 1 in the cosmetics corner (cosmeticssales floor) for 25 seconds, and as a result, contacted with a cosmeticproduct XX (took the cosmetic product XX in hand), and thereafter,stayed in the confectionery corner (confectionery sales floor) for 15seconds, and bought the cosmetic product XX, and that her stay time inthe store was 12 minutes”.

Note that in order for the present signage control system 10 to checkwhether or not, in the behavior of the customer described above, thecustomer has bought the product it has contacted with, the CPU 21(mainly the estimation result linkage circuitry 35) of the analysis box4 compares a face vector estimated based on the frame images from thesignage 1 placed on the product shelf, which places the product thecustomer has contacted with, and a face vector estimated based on theframe images from the signage 1 placed in front of the POS register 7(which has the built-in camera 2 capable of capturing the customerbuying the product), so as to find the timing when a customer havingcontacted with a specific product pays for such product. Further,whether a specific product contacted by a customer is included in theproducts bought by the customer at the time of payment is determinedalso by the CPU 21 by comparing the product, which is captured in theframe images from the signage 1 placed on the product shelf and whichhas been contacted by the customer, and the products which have beensubjected to bar code scanning by the POS register 7.

As described above, the signage control system 10 of the presentexemplary embodiment can track not only the attributes of each customerhaving arrived at a position where such customer can visually recognizea content on the signage 1, and the viewer rating of each advertisingcontent displayed on the signage 1, but also the behavior of suchcustomer, including whether or not such customer has contacted with aproduct after viewing the advertising content displayed on the signage1, and whether or not such customer has bought the product it hascontacted with. Thus, the present signage control system 10 canintroduce an affiliate (performance-based or result reward-type)advertising system, such as Web advertising, for advertising contents tobe displayed on a signage 1 placed in a real store.

As described in the foregoing, according to the signage control system10 and the signage control program (signage-side control program 50shown in FIG. 5 and learned DNN models for various inference processesshown in FIG. 3 ) of the present exemplary embodiment, an advertisingcontent displayed on a signage 1 is changed to another based on theattributes (gender and age or generation) of a customer who is expectedto be in a position where such customer can visually recognize thecontent on the signage 1, and also based on preceding behavior of suchcustomer before then, all of which are contained in a group ofestimation results generated by linking estimation results based onframe images from multiple ones of the cameras (built-in cameras 2 ofthe signages 1 and fixed cameras 3) for the same customer.

Thus, the advertising content displayed on the signage 1 can be changedto another, considering not only the attributes of a customer expectedto be in a position where such customer can visually recognize thecontent on the signage 1, but also the preceding behavior of suchcustomer before such customer has come in front of the signage 1 (to aposition where such customer can visually recognize the content on thesignage 1). Therefore, as compared with the conventional signage controlsystem disclosed in Japanese Laid-open Patent Publication 2020-160780,which changes an advertising content on a signage 1 to another basedonly on the attributes and behavior of a customer analyzed based on theframe images of the customer captured by a signage-side camera, it ispossible to display an advertising content which better matches orcorresponds to the customer in front of the signage 1 (in a positionwhere the customer can visually recognize the content on the signage 1).

Further, in contrast to the conventional signage control systemdisclosed in Japanese Laid-open Patent Publication 2020-160780, whichestimates the attributes and behavior of a customer from only the frameimages (signage-side images) of the customer or other person captured bya signage-side camera, the signage control system 10 of the presentexemplary embodiment is designed to use not only the signage-side imagesbut also frame images (fixed camera-side images) of a customer capturedby a plurality of fixed cameras 3 to estimate the face vector,attributes and behavior of such customer captured in these frame images.Thus, in contrast to the conventional signage control system disclosedin Japanese Laid-open Patent Publication 2020-160780, the process ofestimating the attributes, behavior and the like of a customer who isexpected to be in a position where such customer can visually recognizea content on a signage 1 can be started by using the fixed camera-sideimages having been captured prior to the signage-side images, beforesuch customer comes into the position where such customer can visuallyrecognize the content on the signage 1. Therefore, it is possible toimmediately display a content to attract the interest of such customerwhen such customer comes into a position where such customer canvisually recognize the content on the signage 1.

Further, according to the signage control system 10 of the presentexemplary embodiment, for each customer who is expected to arrive at aposition where such customer can visually recognize a content on thesignage 1 as a result of the estimation using the arrival timeestimation circuitry 34, the content change circuitry 36 operates sothat, at a time point based on the estimated arrival time of suchcustomer at which such customer is expected to arrive at the positionwhere such customer can visually recognize the content on the signage 1as estimated by the arrival time estimation circuitry 34, the contentchange circuitry 36 changes the content displayed on the signage 1 toanother based on the attributes and the preceding behavior of suchcustomer which are contained in a group of estimation results generatedby the estimation result linkage circuitry 35.

Thus, at a time point based on the estimated arrival time of each suchcustomer at which such customer is expected to arrive at the positionwhere such customer can visually recognize the content on the signage 1(for example, at the time point of the estimated arrival time itself, orat a time point 5 seconds after the estimated arrival time), the contentchange circuitry 36 can change the content on the signage 1 to anadvertising content which corresponds to the attributes of such customerexpected to be in the position where such customer can visuallyrecognize the content on the signage 1, and corresponds to the precedingbehavior of such customer before then, and which thus matches suchcustomer. Therefore, it is possible to display an advertising content atthe expected timing of arrival of a customer at a position where suchcustomer can visually recognize the content on the signage 1, making itpossible to surely arouse the interest of such customer.

Further, according to the signage control system 10 of the presentexemplary embodiment, the estimation result linkage circuitry 35 usesthe face vector contained in each estimation result stored in thestorage device 33 for each customer, who is expected to arrive at aposition where such customer can visually recognize a content on eachsignage 1 as a result of the estimation using the arrival timeestimation circuitry 34, to link the estimation results based on theframe images from multiple ones of the cameras (built-in cameras 2 ofthe signages 1 and the fixed cameras 3) for the same customer so as togenerate a group of estimation results for each customer. Thus, thetarget of the estimation result linkage process using the estimationresult linkage circuitry 35 can be narrowed down to the above-describedestimation results for each customer expected to arrive at the positionwhere such customer can visually recognize the content on the signage 1.Therefore, it is possible to reduce the load of the process of the CPU21 and the (inference) chips 24 a to 24 h of the analysis box 4.

Further, the signage control system 10 of the present exemplaryembodiment is designed so that if the number of customers, at a specifictime point, who are expected to be in the position where such customerscan visually recognize the content on the signage 1 is determined to beplural, an advertising content matching or corresponding to a commonattribute for these plural customers in the attributes estimated by theprimary estimation circuitries (primary estimation circuitry 32 of theanalysis box 4 and primary estimation circuitry 41 of the signage 1) forthese plural customers is displayed on the signage 1. Thus, if thenumber of customers, at a specific time point, who are expected to be inthe position where such customers can visually recognize the content onthe signage 1 is plural, an advertising content optimized for all thesecustomers, not an advertising content matching a specific customer, canbe displayed, and therefore, it is possible to protect the privacy ofeach of these customers.

Further, the signage control system 10 of the present exemplaryembodiment is designed to display a predetermined standard content onthe signage 1 if the number of customers, at a specific time point, whoare expected to be in the position where such customers can visuallyrecognize the content on the signage 1 is determined to be plural, andif a common attribute for these plural customers is absent in theattributes estimated by the primary estimation circuitries (primaryestimation circuitry 32 of the analysis box 4 and primary estimationcircuitry 41 of the signage 1) for these plural customers. Thus, it ispossible to protect the privacy of each of these customers.

MODIFIED EXAMPLES

It is to be noted that the present invention is not limited to theabove-described exemplary embodiment, and various modifications arepossible within the spirit and scope of the present invention. Modifiedexamples of the present invention will be described below.

Modified Example 1

The exemplary embodiment described above has shown an example, in whichthe signage 1 is of a tablet terminal type. However, the signage whichcan be used in the present invention is not limited to this, and can beformed by connecting a USB-connectable Web camera and a HDMI(High-Definition Multimedia Interface)-connectable display to a STB (SetTop Box) with communication function. This makes it possible to applythe signage control system of the present invention to a signage controlsystem using a large size signage, and a signage control system using asignage of a various size.

Modified Example 2

In the exemplary embodiment described above, the arrival time estimationcircuitry 34 is designed so that from the motion vector for eachcustomer captured in the fixed camera-side images, and from the timepoint at which such customer appears in the fixed camera-side images,the arrival time estimation circuitry 34 estimates an arrival time ofsuch customer at a position where such customer can visually recognize acontent on a signage 1. However, the arrival time estimation circuitrywhich can be used in the present invention is not limited to this, andcan be designed so that, for example, from motion tracking of eachcustomer (combinations of the center points and time points of boundingboxes for such customer) as estimated by the signage-side primaryestimation circuitry, and from motion tracking of such customer asestimated by the analysis box-side primary estimation circuitry, thearrival time estimation circuitry estimates the arrival time of suchcustomer at a position where such customer can visually recognize thecontent on the signage.

Modified Example 3

Further, in the exemplary embodiment described above, if the number ofcustomers, at a specific time point, who are expected to be in theposition where such customers can visually recognize the content on thesignage 1 is plural, and if there is a common attribute for these pluralcustomers in the attributes estimated by the primary estimationcircuitries for these plural customers, an advertising content matchingor corresponding to the common attribute for these plural customers isdisplayed on the signage 1, while a predetermined standard content isdisplayed on the signage 1 if a common attribute for these pluralcustomers is absent.

However, the advertising content change method which can be used in thepresent invention is not limited to this, and can be designed tounconditionally display a predetermined standard content on the signageif the number of customers, at a specific time point, who are expectedto be in the position where such customers can visually recognize thecontent on the signage is plural. The advertising content change methodcan also be designed so that if the number of customers, at a specifictime point, who are expected to be in the position where such customerscan visually recognize an advertising content on the signage is plural,the method finds a customer in these plural customers who has been forthe longest time (stay time) in the position where such customer canvisually recognize the advertising content on the signage, or who hasbeen viewing the advertising content for the longest time, so as todisplay, on the signage 1, an advertising content matching orcorresponding to the attributes and preceding behavior of such customer.

Modified Example 4

Further, in the exemplary embodiment described above, the signage 1-sideprimary estimation circuitry 41 is designed to estimate, based on thesignage-side images, a face vector for identifying a customer in thesignage-side images, and the attributes and behavior of such customercaptured in the signage-side images. However, the present invention isnot limited to this, and can be designed so that the analysis box-sideprimary estimation circuitry not only performs the estimation process ofthe face vector, attributes and (customer) behavior of a customercaptured in the fixed camera-side images, but also performs theestimation process of the face vector, attributes and behavior of acustomer captured in the signage-side images.

Modified Example 5

Further, the exemplary embodiment described above has shown an example,in which the person feature is a face vector obtained by vectorizing aface image of a customer. However, the person feature in the presentinvention is not limited to this, and can be a customer vector obtainedby vectorizing an image of the entire body of a customer, or can be anyfeature of the face or body of a customer (for example, an outline of aface, a texture of a face such as spots, wrinkles and sagging, adistance between eyes, and so on).

Modified Example 6

Further, in the exemplary embodiment described above, the analysis box 4is designed to comprise the video input circuitry 31 and the primaryestimation circuitry 32. However, the analysis box 4 is not limited tothis, and can be designed so that an AI (Artificial Intelligence) camerawith so-called edge computing function is used for the camera placed ineach store, and an application package comprising learned DNN models forinference processes such as face detection process, attribute estimation(face recognition) process, face vector extraction process, behaviorestimation process is installed on the AI camera so as to allow the AIcamera to have the functions of the video input circuitry and theprimary estimation circuitry described above.

Modified Example 7

The exemplary embodiment described above has shown an example, in whichthe signage control system 10 comprises only the signage managementserver 9 and the not shown POS server on cloud C. However, the signagecontrol system can comprise another server on cloud C. For example, thesignage control system can comprise, on cloud C, a management server tomanage a number of analysis boxes placed in each store, and fixedcameras connected to these analysis boxes, or comprise an AI analysisserver to convert, for output, information on analysis results from theanalysis box to data to facilitate the use of applications for varioususes such as marketing, crime prevention and so on.

These and other modifications will become obvious, evident or apparentto those ordinarily skilled in the art, who have read the description.Accordingly, the appended claims should be interpreted to cover allmodifications and variations which fall within the spirit and scope ofthe present invention.

The invention claimed is:
 1. A signage control system comprising asignage, a signage-side camera to capture images in front of the signageand at least one surveillance camera to capture a given capture area,wherein the signage control system further comprises: a primaryestimation circuitry configured to use signage-side images which areframe images from the signage-side camera and surveillance-side imageswhich are frame images from the at least one surveillance camera so asto estimate a person feature of each person in these frame images, andalso estimate attributes and behavior of the each person captured inthese frame images; a storage device configured to associate and storeresults of estimations of the person feature, the attributes and thebehavior of each specific person as estimated by the primary estimationcircuitry using the frame images from each specific one of thesignage-side cameras and the at least one surveillance camera; anestimation result linkage circuitry configured to use the person featurestored in the storage device to link the results of estimations based onthe frame images from multiple ones of the cameras for the same personso as to generate a group of estimation results for each person; and acontent change circuitry configured to change a content displayed on thesignage to another based on the attributes of each person who isexpected to be in a position where such person can visually recognizethe content on the signage, and also based on preceding behavior of suchperson before then, the attributes and the preceding behavior beingcontained in the group of estimation results generated by the estimationresult linkage circuitry, wherein from a motion vector of each personcaptured in the surveillance-side images, and from a time point at whichsuch person appears in the surveillance-side images, arrival timeestimation circuitry estimates an arrival time of such person at whichsuch person is expected to arrive at the position where such person canvisually recognize the content on the signage.
 2. The signage controlsystem according to claim 1, wherein the arrival time estimationcircuitry is configured to estimate the arrival time of each personcaptured in the frame images captured by the at least one surveillancecamera at which such person is expected to arrive at the position wheresuch person can visually recognize the content on signage, wherein foreach person who is expected to arrive at the position where such personcan visually recognize the content on the signage as a result of theestimation using the arrival time estimation circuitry, the contentchange circuitry operates so that, at the time point based on estimatedarrival time of such person at which such person is expected to arriveat the position where such person can visually recognize the content onthe signage as estimated by the arrival time estimation circuitry, thecontent change circuitry changes the content displayed on the signage toanother based on the attributes and the preceding behavior of suchperson which are contained in the group of estimation results generatedby the estimation result linkage circuitry.
 3. The signage controlsystem according to claim 2, wherein based on the person feature storedin the storage device for each person who is expected to arrive at theposition where such person can visually recognize the content as aresult of the estimation using the arrival time estimation circuitry,the estimation result linkage circuitry links the estimation resultsbased on the frame images from the multiple ones of the cameras for thesame person so as to generate the group of estimation results for eachperson.
 4. The signage control system according to claim 2, wherein ifthe number of persons, at a specific time, who are expected to be in theposition where such persons can visually recognize the content on thesignage is determined to be plural as a result of the estimation usingthe arrival time estimation circuitry, the content change circuitrydisplays, on the signage, a content matching a common attribute forthese plural persons in the attributes estimated by the primaryestimation circuitry for these plural persons.
 5. The signage controlsystem according to claim 4, wherein the content change circuitrydisplays a predetermined standard content on the signage if the numberof persons, at a specific time point, who are expected to be in theposition where such persons can visually recognize the content on thesignage is determined to be plural as a result of the estimation usingthe arrival time estimation circuitry, and if a common attribute forthese plural persons is absent in the attributes estimated by theprimary estimation circuitry for these plural persons.
 6. The signagecontrol system according to claim 1, wherein the attributes are genderand age.
 7. A non-transitory computer-readable recording medium forrecording a signage control program to cause a computer to execute aprocess including the steps of: using signage-side images which areframe images from a signage-side camera and surveillance-side imageswhich are frame images from at least one surveillance camera so as toestimate a person feature of each person in these frame images, and alsoestimate attributes and behavior of the each person captured in theseframe images; associating and storing results of estimations of theperson feature, the attributes and the behavior of each specific personusing the frame images from each specific one of the signage-sidecameras and the at least one surveillance camera; using the personfeature stored in a storage device to link the results of estimationsbased on the frame images from multiple ones of the cameras for the sameperson so as to generate a group of estimation results for each person;changing a content displayed on the signage to another based on theattributes of each person who is expected to be in a position where suchperson can visually recognize the content on the signage, and also basedon preceding behavior of such person before then, the attributes and thepreceding behavior being contained in the group of estimation results;and estimating an arrival time of such person at which such person isexpected to arrive at the position where such person can visuallyrecognize the content on the signage, from a motion vector of eachperson captured in the surveillance-side images, and from a time pointat which such person appears in the surveillance-side images.